Image processing system, information processing device, information storage device, and image processing method

ABSTRACT

An image processing system includes an image acquisition section that consecutively acquires still images, a direction determination section that determines a camera moving direction, a slit image generation section that generates slit images based on the consecutively acquired still images, and a panoramic image generation section that performs a synthesis process of the slit images to generate a panoramic image. The direction determination section determines whether the camera moving direction is a first camera moving direction or a second camera moving direction when the imaging section is moved during capture. The panoramic image generation section performs the synthesis process of each slit image based on the camera moving direction when the consecutively acquired still images were captured to generate the panoramic image.

CROSS REFERENCE TO RELATED APPLICATION

This application is a continuation of International Patent ApplicationNo. PCT/JP2012/068057, having an international filing date of Jul. 17,2012, which designated the United States, the entirety of which isincorporated herein by reference. Japanese Patent Application No.2011-164062 filed on Jul. 27, 2011 is also incorporated herein byreference in its entirety.

BACKGROUND

The present invention relates to an image processing system, aninformation processing device, an information storage device, an imageprocessing method, and the like.

In recent years, a technique that generates a panoramic image using adigital camera has been developed. A panoramic image is normallygenerated by stitching a plurality of still images captured while movingan imaging section in one direction.

JP-A-2010-28764 discloses a technique that generates a panoramic imagebased on slits extracted from consecutively captured images to increasethe consecutive shooting speed and the slit synthesis speed whileeliminating discontinuity during image synthesis.

Japanese Patent No. 3466493 discloses a method that generates apanoramic image that is mainly used for navigation applications. Themethod disclosed in Japanese Patent No. 3466493 extracts slits fromconsecutive images captured using an on-board stationary camera tosynthesize a panoramic image. The method disclosed in Japanese PatentNo. 3466493 makes it unnecessary to employ an image synthesis processbased on a feature point that increases the processing load, and cangenerate a panoramic image at a higher speed.

According to the panoramic synthesis method disclosed in JP-A-2010-28764or Japanese Patent No. 3466493, a panoramic image can be synthesizedtaking account of motion parallax by stitching trimmed rectangular areas(slits).

SUMMARY

According to one aspect of the invention, there is provided an imageprocessing system comprising:

an image acquisition section that consecutively acquires still images;

a direction determination section that determines a camera movingdirection, the camera moving direction being a moving direction of animaging section during capture;

a slit image generation section that generates slit images based on theconsecutively acquired still images; and

a panoramic image generation section that performs a synthesis processof the slit images to generate a panoramic image,

the direction determination section determining whether the cameramoving direction is a first camera moving direction or a second cameramoving direction that differs from the first camera moving directionwhen the imaging section is moved during capture, and

the panoramic image generation section determining a synthesis positionof a slit image among the slit images based on the camera movingdirection when the consecutively acquired still images were captured,and performing the synthesis process of the slit image to generate thepanoramic image.

According to another aspect of the invention, there is provided aninformation processing device comprising:

a direction determination section that determines a camera movingdirection, the camera moving direction being a moving direction of animaging section during capture;

a panoramic image generation section that performs a synthesis processof slit images generated based on still images consecutively acquired byan image acquisition section to generate a panoramic image; and

a storage section that stores the slit images and the generatedpanoramic image,

the direction determination section determining whether the cameramoving direction is a first camera moving direction or a second cameramoving direction that differs from the first camera moving directionwhen the imaging section is moved during capture, and

the panoramic image generation section determining a synthesis positionof a slit image among the slit images generated based on theconsecutively acquired still images, based on the camera movingdirection when the consecutively acquired still images were captured,and performing the synthesis process of the slit image to generate thepanoramic image.

According to another aspect of the invention, there is provided acomputer-readable storage device with an executable program storedthereon, wherein the program instructs a computer to perform steps of:

determining a camera moving direction, the camera moving direction beinga moving direction of an imaging section during capture;

performing a synthesis process of slit images generated based onconsecutively acquired still images to generate a panoramic image;

determining whether the camera moving direction is a first camera movingdirection or a second camera moving direction that differs from thefirst camera moving direction when the imaging section is moved duringcapture;

determining a synthesis position of a slit image among the slit imagesgenerated based on the consecutively acquired still images based on thecamera moving direction when the consecutively acquired still imageswere captured; and

performing the synthesis process of the slit image to generate thepanoramic image.

According to another aspect of the invention, there is provided an imageprocessing method comprising:

consecutively acquiring still images;

determining whether a camera moving direction is a first camera movingdirection or a second camera moving direction that differs from thefirst camera moving direction when an imaging section is moved duringcapture, the camera moving direction being a moving direction of theimaging section during capture;

generating slit images based on the consecutively acquired still images;

determining a synthesis position of a slit image among the slit imagesbased on the camera moving direction when the consecutively acquiredstill images were captured; and

performing a synthesis process of the slit image to generate a panoramicimage.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates a system configuration example according to oneembodiment of the invention.

FIG. 2 illustrates a detailed system configuration example according toone embodiment of the invention.

FIGS. 3A to 3C are views illustrating a difference in perspective due tomotion parallax.

FIGS. 4A and 4B are views illustrating a slit image.

FIG. 5 is a view illustrating a technique that generates a panoramicimage while moving an imaging section in a plurality of directions.

FIG. 6 is a view illustrating an overlapping area.

FIGS. 7A to 7C are views illustrating a positioning slit.

FIG. 8 is a view illustrating a method that synthesizes a slit imageusing a positioning slit.

FIGS. 9A and 9B are views illustrating a method that extracts aplurality of positioning slits from a single still image.

FIG. 10 is a view illustrating a method that changes a positioning slitextraction position.

FIGS. 11A and 11B illustrate examples of a method that selects apositioning slit.

FIGS. 12A and 12B illustrate further examples of a method that selects apositioning slit.

FIGS. 13A to 13C are views illustrating a method that limits anoverlapping area search range.

FIG. 14 is a view illustrating a method that increases an imageacquisition rate when a camera moving speed has increased.

FIG. 15 is a view illustrating a method that increases a slit imagewidth when a camera moving speed has increased.

FIG. 16 is a flowchart illustrating the flow of a process according toone embodiment of the invention.

FIG. 17 is a flowchart illustrating the flow of a panoramic synthesisprocess according to one embodiment of the invention.

FIG. 18 illustrates another system configuration example according toone embodiment of the invention.

DESCRIPTION OF EXEMPLARY EMBODIMENTS

JP-A-2010-28764 and Japanese Patent No. 3466493 disclose a techniquethat generates a panoramic image by moving the imaging section in onedirection, but do not disclose a technique that generates a panoramicimage by moving the imaging section in a plurality of directions.

Several embodiments of the invention may provide an image processingsystem, an information processing device, an information storage device,an image processing method, and the like that can generate a panoramicimage based on a slit image when the imaging section is moved in aplurality of directions during capture.

Several embodiments of the invention may provide an image processingsystem, an information processing device, an information storage device,an image processing method, and the like that can generate a panoramicimage based on a slit image and a positioning slit when the imagingsection is moved in a plurality of directions during capture.

According to the above configuration, when the imaging section is movedin a plurality of camera moving directions during capture, still imagesused to generate the slit images are consecutively acquired, and thecurrent camera moving direction is determined. The slit image isgenerated based on a still image among the consecutively acquired stillimages. The synthesis position of the slit image is determined based onthe camera moving direction, and the synthesis process is thenperformed. It is possible to generate a panoramic image of which theangle of view is wider in a plurality of directions than that of apanoramic image generated by moving the imaging section in one cameramoving direction while reducing a difference in perspective due tomotion parallax by utilizing the slit images.

The image processing system may further comprise:

a positioning slit generation section that generates a positioning slitbased on a still image among the consecutively acquired still images,

the panoramic image generation section may determine the synthesisposition of the slit image based on the camera moving direction when theconsecutively acquired still images were captured, and the positioningslit, and may perform the synthesis process of the slit image togenerate the panoramic image.

According to the above configuration, since a problem does not occureven when one slit image has an overlapping area with a plurality ofslit images, the overlapping area can be relatively easily specified,for example.

In the image processing system,

the positioning slit generation section may generate the positioningslit so that a long side of the slit image is orthogonal to a long sideof the positioning slit.

According to the above configuration, the overlapping area can bespecified so that the short side of the slit image coincides with thelong side of the positioning slit, for example.

In the image processing system,

the positioning slit generation section may set a first area and asecond area based on a straight line that passes through a center of thestill image, may generate a first positioning slit from the first area,and may generate a second positioning slit from the second area.

The above configuration makes it possible to selectively use the firstpositioning slit or the second positioning slit when specifying theoverlapping area of the slit images, for example. Specifically, it ispossible to select the positioning slit corresponding to the cameramoving direction, and determine the overlapping area even when theimaging section is moved in an arbitrary direction, for example.

In the image processing system,

the positioning slit generation section may change an extractionposition of the positioning slit within the still image based on motioninformation about the imaging section acquired from a motion informationacquisition section, and may generate the positioning slit.

The above configuration makes it possible to extract the positioningslit from the same position as that of the previously used positioningslit when the position of the imaging section has changed to a largeextent, and efficiently perform the search process that searches theoverlapping area of the synthesis target image and the positioning slit,for example.

In the image processing system,

the panoramic image generation section may shift the synthesis positionin a direction perpendicular to the first camera moving direction whenit has been determined that the camera moving direction is the secondcamera moving direction, and may perform the synthesis process so thatpart of the positioning slit overlaps part of the slit image to generatethe panoramic image.

The above configuration makes it possible to generate a panoramic imagethat covers a wide range of the imaging area in the vertical directionand the horizontal direction when the imaging section was moved along azigzag path, for example.

In the image processing system,

the direction determination section may determine the camera movingdirection when the imaging section is moved in the first camera movingdirection, moved in a third camera moving direction that differs fromthe first camera moving direction and the second camera movingdirection, and then moved in the second camera moving direction duringcapture, and

the panoramic image generation section may use the positioning slitgenerated based on the still image acquired when the camera movingdirection is the first camera moving direction for the synthesis processwhen the positioning slit was generated from an area of the still imageassigned in the third camera moving direction relative to a center ofthe still image, and may use the positioning slit generated based on thestill image acquired when the camera moving direction is the secondcamera moving direction for the synthesis process when the positioningslit was generated from an area of the still image assigned in adirection opposite to the third camera moving direction relative to thecenter of the still image.

The above configuration makes it possible to specify the positioningslit by which the slit image can be synthesized so that the angle ofview of the synthesized image is sufficiently larger than the angle ofview of the original slit image.

In the image processing system,

the panoramic image generation section may limit a search range for aposition of an overlapping area of the positioning slit and the slitimage based on motion information about the imaging section acquiredfrom a motion information acquisition section, and may perform thesynthesis process.

The above configuration makes it possible to limit the overlapping areasearch range to an area around the position estimated based on themotion information after the imaging section has been moved, and performthe process that specifies the overlapping area at high speed (i.e.,reduce the load of the process that specifies the overlapping area), forexample.

In the image processing system,

the panoramic image generation section may limit the search range forthe position of the overlapping area of the positioning slit and theslit image based on a horizontal component or a vertical component ofthe motion information about the imaging section acquired from themotion information acquisition section, and may perform the synthesisprocess.

According to the above configuration, the search range can be limitedrelative to the positioning slit.

In the image processing system,

the slit image generation section may generate the slit images that havea long side along a direction corresponding to a horizontal component ora vertical component, whichever is smaller in absolute value, of avector that indicates the camera moving direction.

The above configuration makes it possible to generate a slit image thatdoes not include an area of the still image in the camera movingdirection and an area of the still image in the direction opposite tothe camera moving direction, and suppress the effects of a difference inperspective due to motion parallax, for example.

In the image processing system,

the slit image generation section may generate the slit images that havethe long side along the direction corresponding to the horizontalcomponent or the vertical component, whichever is smaller in absolutevalue, of the vector that indicates the camera moving direction, andinclude a center of a still image among the consecutively acquired stillimages.

The above configuration makes it possible to generate a slit image thatdoes not include an area of the still image in the camera movingdirection and an area of the still image in the direction opposite tothe camera moving direction, and includes the center of the still image,and further suppress the effects of a difference in perspective due tomotion parallax, for example.

In the image processing system,

the panoramic image generation section may perform the synthesis processof the slit image so that the slit image has an overlapping area withthe slit image that was used for a preceding synthesis process in thecamera moving direction when the slit image used for the precedingsynthesis process was captured to generate the panoramic image.

The above configuration makes it possible to perform the synthesisprocess while making the generated slit images overlap in the cameramotion direction.

In the image processing system,

the direction determination section may determine whether the cameramoving direction is the first camera moving direction, the second cameramoving direction, or a third camera moving direction that differs fromthe first camera moving direction and the second camera moving directionwhen the imaging section is moved during capture.

The above configuration makes it possible to generate the panoramicimage by moving the imaging section along a zigzag path.

In the image processing system,

the panoramic image generation section may perform a process thatsynthesizes the slit images in the first camera moving direction and aprocess that synthesizes the slit images in the second camera movingdirection in parallel.

According to the above configuration, since the slit images can besynthesized in the second camera moving direction even when the slitimages have not been completely synthesized in the first camera movingdirection, it is possible to prevent a situation in which the processthat synthesizes the slit images in the first camera moving directionbecomes a bottleneck, and the panoramic image generation process can beperformed at high speed, for example.

In the image processing system,

the panoramic image generation section may perform the synthesis processof the slit image based on a first error tolerance, and may then performthe synthesis process of the slit image based on a second errortolerance that is lower than the first error tolerance.

According to the above configuration, the overlapping area of the slitimage can be specified based on the first error tolerance, and theaccuracy of the synthesis position of the slit image can be improvedbased on the second error tolerance, for example.

In the image processing system,

the image acquisition section may stop acquisition of the still imageswhen motion information about the imaging section has been acquired frommotion information acquisition section, and it has been determined thatan overlapping area of consecutive slit images among the slit images hasnot occurred in the synthesis process of the slit image based on themotion information.

The above configuration makes it possible to prevent a situation inwhich the still images are consecutively acquired although consecutiveslit images do not have an overlapping area, and a panoramic imagecannot be generated, for example.

In the image processing system,

the image acquisition section may increase an image acquisition rate sothat an overlapping area of consecutive slit images among the slitimages is larger than a given threshold value during the synthesisprocess of the slit image when it has been determined that a cameramoving speed has increased based on motion information about the imagingsection acquired from a motion information acquisition section, thecamera moving speed being a moving speed of the imaging section.

The above configuration makes it possible to obtain the slit imageshaving an overlapping area necessary for synthesizing the panoramicimage.

In the image processing system,

the slit image generation section may increase a width of the slit imageso that an overlapping area of consecutive slit images among the slitimages is larger than a given threshold value during the synthesisprocess of the slit image when it has been determined that a cameramoving speed has increased based on motion information about the imagingsection acquired from a motion information acquisition section, thecamera moving speed being a moving speed of the imaging section.

The above configuration makes it possible to obtain the slit imageshaving an overlapping area necessary for synthesizing the panoramicimage even when the image acquisition rate (sampling rate) cannot beincreased due to limitations to the hardware or the network, forexample.

In the image processing system,

the image acquisition section may control a shutter release timing ofthe imaging section to consecutively acquire the still images.

The above configuration makes it possible to cause the number of stillimages captured by the imaging section to coincide with the number ofstill images acquired by the image acquisition section, and cause theimage acquisition section to acquire the desired number of still imageswhile preventing a situation in which the imaging section capturesunnecessary still images, for example.

In the information processing device,

the storage section may store a positioning slit generated by apositioning slit generation section based on a still image among theconsecutively acquired still images, and

the panoramic image generation section may determine the synthesisposition of the slit image based on the camera moving direction when theconsecutively acquired still images were captured, and the positioningslit, and may perform the synthesis process of the slit image togenerate the panoramic image.

In the information processing device,

the panoramic image generation section may shift the synthesis positionin a direction perpendicular to the first camera moving direction whenit has been determined that the camera moving direction is the secondcamera moving direction, and may perform the synthesis process so thatpart of the positioning slit overlaps part of the slit image to generatethe panoramic image.

In the information processing device,

the slit images may have a long side along a direction corresponding toa horizontal component or a vertical component, whichever is smaller inabsolute value, of a vector that indicates the camera moving direction,and may include a center of a still image among the consecutivelyacquired still images.

In the information processing device,

the panoramic image generation section may perform the synthesis processof the slit image so that the slit image has an overlapping area withthe slit image that was used for a preceding synthesis process in thecamera moving direction when the slit image used for the precedingsynthesis process was captured to generate the panoramic image.

Exemplary embodiments of the invention are described below. An outlineand a system configuration example will be described first. A specificembodiment and a method will then be described. The flow of a processwill then be described using a flowchart. Note that the followingexemplary embodiments do not in any way limit the scope of the inventionlaid out in the claims. Note also that all of the elements describedbelow in connection with the following exemplary embodiments should notnecessarily be taken as essential elements of the invention.

1. Outline

In recent years, a technique that generates a panoramic image using adigital camera has been developed. A panoramic image is normallygenerated by stitching a plurality of still images captured while movingan imaging section in one direction.

When an image is captured while moving the imaging section, a differencein perspective due to motion parallax may occur in the captured stillimages (see FIGS. 3A to 3C). In this case, the still images may not besuccessfully synthesized due to the difference in perspective, and itmay be difficult to generate a panoramic image. In order to reduce theeffects of a difference in perspective, a technique that extracts a slitimage from a still image, and generates a panoramic image using the slitimage has been developed. For example, JP-A-2010-28764 and JapanesePatent No. 3466493 disclose such a technique.

JP-A-2010-28764 discloses a technique that generates a panoramic imagebased on slits extracted from consecutively captured images to increasethe consecutive camera moving speed and the slit synthesis speed whileeliminating discontinuity during image synthesis.

Japanese Patent No. 3466493 discloses a method that generates apanoramic image that is mainly used for navigation applications. Themethod disclosed in Japanese Patent No. 3466493 extracts slits fromconsecutive images captured using an on-board stationary camera tosynthesize a panoramic image. The method disclosed in Japanese PatentNo. 3466493 makes it unnecessary to employ an image synthesis processbased on a feature point that increases the processing load, and cangenerate a panoramic image at a higher speed.

According to the panoramic synthesis method disclosed in JP-A-2010-28764or Japanese Patent No. 3466493, a panoramic image can be synthesizedtaking account of motion parallax by stitching trimmed rectangular areas(slits).

However, JP-A-2010-28764 and Japanese Patent No. 3466493 disclose atechnique that generates a panoramic image by moving the imaging sectionin one direction, but do not disclose a technique that generates apanoramic image by moving the imaging section in a plurality ofdirections.

A technique that generates a panoramic image by moving the imagingsection in one direction has a problem in that the angle of view of thepanoramic image can be adjusted only in the moving direction of theimaging section. For example, the angle of view of the panoramic imagecannot be increased in the direction perpendicular to the movingdirection of the imaging section.

If still images captured while moving the imaging section in a pluralityof directions can be provided by moving the imaging section in aplurality of directions during capture, it is possible to generate apanoramic image of which the angle of view is increased in a pluralityof directions.

According to several embodiments of the invention, when the imagingsection is moved in a plurality of camera moving directions duringcapture, still images are consecutively acquired, slit images areextracted from the consecutively acquired still images, and a panoramicimage is generated based on the extracted slit images.

Since the overlapping area of the slit images is smaller than theoverlapping area of the still images, it may be difficult to specify theoverlapping area of the slit images. According to several embodiments ofthe invention, a positioning slit is extracted from the still image inaddition to the slit image, and a panoramic image is generated based onthe slit image and the positioning slit in order to deal with the aboveproblem.

2. System Configuration Example

FIG. 1 illustrates a configuration example of an image processing systemaccording to one embodiment of the invention. In one embodiment of theinvention, the image processing system is a server on an imageprocessing cloud network. The image processing system may include a headmounted display (HMD), an imaging device (camera), and the like inaddition to the image processing cloud network. The image processingsystem may be an information processing device included in an HMD or thelike.

In one embodiment of the invention, the user captures the object usingan imaging device included in (provided to) an HMD while shaking his/herhead to generate still images (or slit images or a movie), which aretransmitted to the server on the image processing cloud network througha base station via wireless communication. The server generates apanoramic image, and the user views the panoramic image on the HMD.

Note that the embodiments of the invention are for illustrative purposesonly, and the image processing system is not limited to theconfiguration illustrated in FIG. 1. For example, various modificationsmay be made, such as omitting some of the elements illustrated in FIG.1, or adding other elements. Note that the still image or the like neednot necessarily be transmitted to the image processing system viawireless communication, but may be transmitted to the image processingsystem via cable communication. When the image processing system is aninformation processing device that is included in the HMD, the imageprocessing system need not necessarily receive the still image or thelike through a network, but may acquire the still image or the likethrough an internal bus or the like. The imaging device need notnecessarily be integrated with the HMD, but may be a handheld camera orthe like. The output device (e.g., HMD) need not necessarily beprovided.

FIG. 2 illustrates a detailed configuration example of the imageprocessing system according to one embodiment of the invention. An imageprocessing system 200 illustrated in FIG. 2 includes a directiondetermination section 210, a panoramic image generation section 220, astorage section 230, an I/F section 240, a slit image generation section250, a positioning slit generation section 260, and an image acquisitionsection (sampling section) 270. Note that the image processing system200 is not limited to the configuration illustrated in FIG. 2. Variousmodifications may be made, such as omitting some of the elementsillustrated in FIG. 2, or adding other elements. For example, the imageprocessing system 200 may be implemented by a plurality of informationprocessing devices.

The image processing system 200 is connected to an imaging device 100and a presentation section 300. Examples of the imaging device 100include a camera included in an HMD and the like. Examples of the imageprocessing system 200 include a server on an image processing cloudnetwork, and the like. Examples of the presentation section 300 includean HMD and the like.

The imaging device 100 includes an imaging section 12, an encoder 14, amotion information acquisition section 16, and an I/F section 18. Notethat the imaging device 100 is not limited to the configurationillustrated in FIG. 2. Various modifications may be made, such asomitting some of the elements illustrated in FIG. 2, or adding otherelements.

The connection relationship between the above sections is describedbelow. In the imaging device 100, the imaging section 12, the encoder14, the motion information acquisition section 16, and the I/F section18 are connected through an internal bus. In the image processing system200, the direction determination section 210, the panoramic imagegeneration section 220, the storage section 230, the I/F section 240,the slit image generation section 250, the positioning slit generationsection 260, and the image acquisition section (sampling section) 270are connected through an internal bus.

A process performed by each section of the imaging device 100 isdescribed below.

The imaging section (camera) 12 captures the object. The imaging section12 includes an image sensor (e.g., CCD) and an optical system. Theimaging section 12 may include a device (processor) that is used forimage processing and the like.

The encoder 14 encodes a movie captured by the imaging section 12 usinga video codec (e.g., MPEG codec). The function of the encoder 14 may beimplemented by hardware such as a processor (e.g., CPU) or an ASIC(e.g., gate array), a program, or the like.

The motion information acquisition section 16 acquires motioninformation about the imaging section 12. The motion informationacquisition section 16 may be a sensor such as an orientation sensor(e.g., terrestrial magnetism sensor), an acceleration sensor, or a gyrosensor. The motion information acquisition section 16 may acquire sensorinformation obtained by the sensor as the motion information about theimaging section 12. The orientation sensor is a terrestrial magnetismsensor or the like, and measures the orientation (angle (0 to 360°)) ofthe sensor. The terrestrial magnetism sensor includes a device thatchanges in resistance value or impedance value depending on the strengthof a magnetic field, for example. The terrestrial magnetism sensordetects triaxial terrestrial magnetism information. The accelerationsensor includes a device that changes in resistance value depending onthe external force, for example. The acceleration sensor detectstriaxial acceleration information. The gyro sensor detects triaxialangular velocity information. A sensor that functions as the terrestrialmagnetism sensor, the acceleration sensor, and/or the gyro sensor mayalso be used. The motion information acquisition section 16 may useposition information obtained by a GPS as the motion information aboutthe imaging section 12.

The motion information acquisition section 16 may acquire the amount ofchange in the imaging range or the like that can be specified from aninternal camera parameter as the motion information about the imagingsection 12. The motion information acquisition section 16 may acquire amotion vector as the motion information, the motion vector beingobtained when the encoder 14 encodes the movie captured by the imagingsection 12. The motion information acquisition section 16 may calculatethe motion information using a tracking algorithm (e.g., optical flowanalysis).

The I/F section 18 notifies the image processing system 200 of theinformation acquired from the imaging section 12, the encoder 14, andthe motion information acquisition section 16.

A process performed by each section of the image processing system 200is described below.

The direction determination section 210 determines a camera movingdirection (described later).

The panoramic image generation section 220 generates a panoramic imagebased on the slit images acquired from the slit image generation section250 (described later).

The storage section 230 stores a database, and serves as a work area forthe panoramic image generation section 220 and the like. The function ofthe storage section 230 may be implemented by a memory (e.g., RAM), ahard disk drive (HDD), or the like. The storage section 230 may storethe still images acquired from the I/F section 240 or the imageacquisition section (sampling section) 270 (described later), the slitimages acquired from the slit image generation section 250, positioningslits acquired from the positioning slit generation section 260, thepanoramic image generated by the panoramic image generation section 220,and the like.

The I/F section 240 handles information communication between the imageprocessing system 200 and the imaging device 100, and informationcommunication between the image processing system 200 and thepresentation section 300. Note that the I/F section 240 may performcommunication via cable communication or wireless communication.

The slit image generation section 250 generates the slit images based onthe still images acquired from the image acquisition section 270(described later).

The positioning slit generation section 260 generates the positioningslits based on the still images acquired from the image acquisitionsection 270 (described later).

The image acquisition section (sampling section) 270 acquires stillimages from the information acquired from the I/F section 240, andoutputs a series of still images to each functional section. The imageacquisition section (sampling section) 270 may include a samplingcontrol section 272.

When the I/F section 240 has acquired information including a moviecaptured by the imaging device 100, the image acquisition section(sampling section) 270 samples still images from the captured movie atan image acquisition rate (sampling rate) set by the sampling controlsection 272, and outputs a series of still images to each functionalsection.

When the I/F section 240 has acquired information including a series ofstill images, the image acquisition section 270 acquires the series ofstill images, and outputs the series of still images to each functionalsection. The series of still images may have been generated by anarbitrary method. For example, the imaging device 100 may haveconsecutively captured (generated) the series of still images, or theimaging device 100 may have captured a movie, and sampled (generated)the series of still images from the movie.

The sampling control section 272 sets the image acquisition rate(sampling rate). When the imaging device 100 consecutively capturesstill images, the sampling control section 272 may output the samplingrate and the like to the I/F section 18 of the imaging device 100 sothat the sampling rate set by the sampling control section 272 can beused as the shutter speed or the capture interval.

The functions of the direction determination section 210, the panoramicimage generation section 220, the slit image generation section 250, thepositioning slit generation section 260, and the image acquisitionsection (sampling section) 270 may be implemented by hardware such as aprocessor (e.g., CPU) or an ASIC (e.g., gate array), a program, or thelike.

The presentation section 300 presents the panoramic image or the likeacquired from the I/F section 240 of the image processing system 200 tothe user. The presentation section 300 may include a sound outputsection and/or a vibration section in addition to the display section.

When the imaging device 100 is a smart camera or the like that has animage processing function, the configuration illustrated in FIG. 18 maybe employed.

As illustrated in FIG. 18, the imaging device 100 may include theimaging section 12, the encoder 14, the motion information acquisitionsection 16, the I/F section 18, the slit image generation section 250,the positioning slit generation section 260, and the image acquisitionsection (sampling section) 270. An information processing device 400 mayinclude the direction determination section 210, the panoramic imagegeneration section 220, the storage section 230, and the I/F section240. Note that the imaging device 100 and the information processingdevice 400 are not limited to the configuration illustrated in FIG. 18.Various modifications may be made, such as omitting some of the elementsillustrated in FIG. 18, or adding other elements.

The function of each section included in the imaging device 100 and theinformation processing device 400 is the same as described above.

According to the configuration illustrated in FIG. 18, since it sufficesfor the imaging device 100 to transmit only the slit images and thepositioning slits to the information processing device 400, the datatransfer amount can be reduced.

3. Specific Embodiment

FIG. 3A illustrates a state in which an object OB is captured whilemoving the imaging section. FIG. 3B illustrates an image IM1 obtainedwhen the object OB is captured from an imaging section CAM1 before theimaging section is moved, and FIG. 3C illustrates an image IM2 obtainedwhen the object OB is captured from an imaging section CAM2 after theimaging section has been moved.

In this case, when the image IM1 and the image IM2 are synthesized togenerate a panoramic image, the areas of the object OB may not besmoothly joined even when the position of the right end of the image IM1corresponds to the position of the left end of the image IM2. Such asituation occurs when the image IM1 and the image IM2 differ inperspective due to motion parallax. Each end of an image is normallyaffected by motion parallax to a large extent as compared with thecenter area of the image.

In one embodiment of the invention, a slit image (SL1) is extracted fromthe sampled still image (SIM1), and synthesized to generate a panoramicimage (see FIG. 4A). As illustrated in FIG. 4B, it is desirable that aslit image (SL2) includes a center point (CP) of the still image (SIM2)in order to reduce the effects of motion parallax. In FIGS. 4A and 4B,the slit image is generated to have a long side along the verticaldirection of the still image. Note that the slit image may be generatedto have a long side along the horizontal direction of the still image.

In one embodiment of the invention, the object is captured while movingthe imaging section in a plurality of directions in order to generate apanoramic image having a wide angle of view in a plurality ofdirections. If the capture direction is completely randomly selected, itmay be difficult to implement positioning, and a smooth panoramic imagemay not be obtained. In order to deal with this problem, the object iscaptured while moving the imaging section CAM along a zigzag path (seeFIG. 5), and a two-dimensional panoramic image PIM is generated from thecaptured movie. In FIG. 5, the object is captured while moving theimaging section CAM in a first camera moving direction DR1 (describedlater), moving the imaging section CAM in a third camera movingdirection DR3 (described later), and then moving the imaging section CAMin a second camera moving direction DR2 (described later). After movingthe imaging section CAM in the second camera moving direction DR2, theobject may be captured while moving the imaging section CAM in anothercamera moving direction.

Note that the process that extracts (samples) the still images used forsynthesis of the panoramic image, the process that extracts the slitimages, and the process that generates the panoramic image may beperformed by the imaging device, or may be performed by an imageprocessing system (e.g., computer) that is connected to the imagingdevice through a communication section via a cable or wireless network.

In one embodiment of the invention, the object is captured withinapproximately the field-of-view range using a head mounted-type imagingsection CAM illustrated in FIG. 5.

A movie that reflects the motion of the user's head when the user looksaround a scene can be easily captured by utilizing the head mounted-typeimaging section CAM, and a realistic panoramic image can be generated.

When generating a panoramic image using the slit images, it is necessaryto perform a process that specifies the overlapping area of the slitimages in the short side direction when synthesizing the slit imageacquired based on the movie when moving the imaging section in the firstcamera moving direction in FIG. 5 and the slit image acquired based onthe movie when moving the imaging section in the second camera movingdirection in FIG. 5. FIG. 6 illustrates a state in which an overlappingarea REA of the slit image SL1 and the slit image SL2 in the short sidedirection is specified.

An area that includes an identical range of the object is normallydetermined to be the overlapping area REA of the slit image SL1 and theslit image SL2 by comparing the slit image SL1 and the slit image SL2(see FIG. 6). However, since the short side of the slit image is muchshorter than the long side of the slit image, the overlapping area REAmay be very small, and it may be difficult to specify the overlappingarea when the position of the slit image SL1 and the position of theslit image SL2 are shifted in the horizontal direction (see FIG. 6). Insuch a case, since one slit image has an overlapping area with two (aplurality of) slit images, it is necessary to compare the slit imagewith two slit images when determining the overlapping area of the slitimage.

The above problem may be solved by synthesizing the slit images in thefirst camera moving direction so that the short side of one slit imageand the short side of another slit image are joined to have a sufficientlength, and then synthesizing the slit images in the second cameramoving direction. Specifically, the upper half (i.e., a partcorresponding to the first camera moving direction DR1) of the panoramicimage PIM is synthesized, and the lower half (i.e., a part correspondingto the second camera moving direction DR2) of the panoramic image PIM isthen synthesized (see FIG. 5). In this case, since a problem in whichone slit image has an overlapping area with a plurality of slit imagesdoes not occur, differing from the example illustrated in FIG. 6, theoverlapping area can be relatively easily specified.

According to the above method, however, the slit images cannot besynthesized in the second camera moving direction when the slit imageshave not been completely synthesized in the first camera movingdirection. For example, the upper half and the lower half of thepanoramic image PIM cannot be synthesized in parallel (see FIG. 5).Therefore, the process that synthesizes the slit images in the firstcamera moving direction may become a bottleneck, and hinder the overallprocess.

In one embodiment of the invention, the slit images are synthesizedusing positioning slits PFSL1 to PFSL3 illustrated in FIGS. 7A to 7C.The positioning slits PFSL1 to PFSL3 are respectively extracted from thestill images SIM1 to SIM3 so that the positioning slits PFSL1 to PFSL3are orthogonal to the slit images. The positioning slit may be extractedat an arbitrary extraction position. For example, the positioning slitmay be extracted at the extraction positions illustrated in FIGS. 7A to7C. The positioning slit may be generated each time the slit image isgenerated, or may be generated when a given number of slit images havebeen generated (i.e., the slit images have been generated a given numberof times). The following description is given taking an example in whichthe positioning slit is generated when a given number of slit imageshave been generated.

In one embodiment of the invention, the slit images are not compareddirectly to determine the overlapping area. As illustrated in FIG. 8,the overlapping area of the slit image SL1 and the positioning slit PFSLis calculated, and the overlapping area of the slit image SL2 and thepositioning slit PFSL is then calculated. The overlapping area (REA3) ofthe slit images SL1 and SL2 can thus be calculated indirectly.

The example illustrated in FIG. 8 is described in detail below. The slitimages SL1 and SL2, and the positioning slit PFSL illustrated in FIG. 8are generated based different still images. Specifically, thepositioning slit PFSL is acquired based on the still image captured whenthe imaging section is moved in the first camera moving direction DR1.The slit image SL1 is generated based on another still image capturedwhen the imaging section is further moved in the first camera movingdirection DR1.

The image included in the slit image SL1 is compared with the imageincluded in the positioning slit PFSL (i.e., image comparison process)to determine the overlapping area REA1. For example, the slit image SL1is assigned at the left end of the positioning slit (initial position).The image included in the slit image SL1 is compared with the imageincluded in the positioning slit PFSL while moving the slit image SL1from the initial position in the rightward direction. When it has beendetermined that the image included in the slit image SL1 coincides withthe image included in the positioning slit PFSL, the coincident area isdetermined to be the overlapping area of the slit image SL1 and thepositioning slit PFSL. When the image included in the slit image SL1does not coincide with the image included in the positioning slit PFSLeven when the slit image SL1 has been moved to the right end of thepositioning slit PFSL, the slit image SL1 is moved in the downwarddirection, and the image included in the slit image SL1 is compared withthe image included in the positioning slit PFSL while moving the slitimage SL1 in the leftward direction. The above process is repeated whilemoving the slit image SL1 in the downward direction and the horizontaldirection. The slit images cannot be synthesized when an overlappingarea cannot be specified by moving the slit image SL1 in the downwarddirection and the horizontal direction. Note that it is unnecessary toperform the above process when the slit image SL1 and the positioningslit PFSL have been extracted from an identical still image since theoverlapping area REA1 is known.

The slit image SL2 is generated based on the still image captured whenthe imaging section is moved in the second camera moving direction DR2.The image included in the slit image SL2 is compared with the imageincluded in the positioning slit PFSL in the same manner as describedabove instead of comparing the image included in the slit image SL2 withthe image included in the slit image SL1 to determine the overlappingarea REA2. In FIG. 8, the overlapping area REA2 is drawn for convenienceof illustration so that the right edge and the left edge of theoverlapping area REA2 do not coincide with the right edge and the leftedge of the slit image SL2. Note that the right edge and the left edgeof the overlapping area REA2 coincide with the right edge and the leftedge of the slit image SL2 in the actual situation.

The overlapping area of the overlapping areas REA1 and REA2 is thencalculated to determine the overlapping area REA3 of the slit images SL1and SL2.

The above method has an advantage in that the overlapping area can beeasily specified (i.e., does not become small) even when the slit imagesSL1 and SL2 are shifted from each other in the horizontal directionsince the overlapping area of the slit image and the positioning slit iscalculated instead of directly calculating the overlapping area of theslit images SL1 and SL2.

Moreover, since the slit images can be synthesized in the second cameramoving direction even when the slit images have not been completelysynthesized in the first camera moving direction, it is possible toprevent a situation in which the process that synthesizes the slitimages in the first camera moving direction becomes a bottleneck, andthe panoramic image generation process can be performed at high speed,for example.

It is desirable to perform the synthesis process in two steps in orderto implement more accurate image synthesis. In the first step, theoverlapping area of the slit images is calculated using the positioningslit to obtain a temporary slit image synthesis position (firstsynthesis process). In the second step, the accuracy of the slit imagesynthesis position is improved to obtain the final slit image synthesisposition (second synthesis process). Note that the second synthesisprocess includes a matching process that utilizes a feature point, or athree-dimensional reconstruction process that utilizes bundle adjustmentor the like.

For example, when the third camera moving direction DR3 (see FIG. 5) hasnot been determined to be the upward direction or the downwarddirection, but it has been determined that the overlapping area isspecified using the positioning slit acquired when the imaging sectionis moved in the first camera moving direction DR1, it may be impossibleto determine the overlapping area as described above using thepositioning slit depending on the extraction position of the positioningslit.

In order to deal with such a situation, a plurality of positioning slitsmay be extracted from a single still image, as illustrated in FIGS. 9Aand 9B. In FIG. 9A, a first positioning slit PFSL1 is extracted from afirst area SUB1 (described later) of the still image SIM1, and a secondpositioning slit PFSL2 is extracted from a second area SUB2 (describedlater) of the still image SIM1. In FIG. 9B, a first positioning slitPFSL3 is extracted from a first area SUB3 (described later) of the stillimage SIM2, and a second positioning slit PFSL4 is extracted from asecond area SUB4 (described later) of the still image SIM2.

The above configuration makes it possible to selectively use the firstpositioning slit or the second positioning slit when specifying theoverlapping area of the slit images, for example. Specifically, it ispossible to select the positioning slit corresponding to the cameramoving direction, and determine the overlapping area even when theimaging section is moved in an arbitrary direction, for example.

For example, when using the positioning slit acquired when the imagingsection is moved in the first camera moving direction in the exampleillustrated in FIG. 5, the first positioning slit PFSL1 illustrated inFIG. 9A can be used when the third camera moving direction is the upwarddirection, and the second positioning slit PFSL2 illustrated in FIG. 9Acan be used when the third camera moving direction is the downwarddirection.

Moreover, when using the positioning slit acquired when the imagingsection is moved in the first camera moving direction, the positionalrelationship between the positioning slit and the slit image acquiredwhen the imaging section is moved in the first camera moving directioncan be determined before moving the imaging section in the second cameramoving direction, and the process that specifies the overlapping areacan be started immediately after the imaging section has been moved inthe second camera moving direction, and the slit image has beenacquired. This makes it possible to reduce the processing time, forexample.

FIG. 10 illustrates an example in which an image PIM1 is generated whenthe imaging section is moved in the first camera moving direction DR1,and the imaging section is then moved in the second camera movingdirection DR2. The positioning slit is generated based on the stillimage captured when the imaging section is moved in the second cameramoving direction DR2. The overlapping area of the image PIM1 and thepositioning slit is searched in the downward direction.

In this case, when the imaging section has been moved as indicated by amotion vector DRV, and the position of the imaging section has changedfrom a position CP1 to a position CP2, the overlapping area search rangechanges to a large extent in the vertical direction within the imagePIM1 when the positioning slit is extracted from a position PFP, and ittakes time to search the overlapping area in proportion to the magnitudeof the vertical component of the motion vector DRV.

In order to deal with the above situation, the extraction position ofthe positioning slit is dynamically changed based on the motioninformation acquired by the motion information acquisition section. InFIG. 10, the extraction position of the positioning slit within a stillimage SIM2 is changed to extract a positioning slit PFSL2 that hascontinuity with a positioning slit PFSL1. This makes it possible toprevent a situation in which the overlapping area search range changesto a large extent in the vertical direction within the image PIM1.Therefore, it is possible to efficiently perform the overlapping areasearch process that searches the overlapping area of the image PIM1 andthe positioning slit PFSL2.

The positioning slit is extracted from both the still image capturedwhen the imaging section is moved in the first camera moving directionand the still image captured when the imaging section is moved in thesecond camera moving direction. The positioning slit used to specify theoverlapping area may be determined as described below.

FIG. 11A illustrates an example in which the positioning slit isgenerated from the upper part of the still image (i.e., a part of thestill image assigned in the third camera moving direction). In theexample illustrated in FIG. 11A, positioning slits PFSL1 and PFSL2 aregenerated based on the still images sampled when the camera movingdirection is the first camera moving direction DR1, and a positioningslit PFSL3 is generated based on the still image sampled when the cameramoving direction is the second camera moving direction DR2.

In this case, it is necessary to select the positioning slit used whensynthesizing the upper part (e.g., slit image SL3) and the lower part(e.g., slit image SL2) of the panoramic image. Since the third cameramoving direction DR3 is the upward direction in the example illustratedin FIG. 11A, the positioning slits PFSL1 and PFSL2 are assigned at theboundary between the upper half and the lower half of the panoramicimage.

Therefore, the slit images are synthesized using the positioning slitsPFSL1 and PFSL2 generated based on the still images sampled when thecamera moving direction is the first camera moving direction DR1.

FIG. 11B illustrates an example in which the positioning slit isgenerated from the lower part of the still image (i.e., a part of thestill image assigned in the third camera moving direction). In theexample illustrated in FIG. 11B, positioning slits PFSL4 and PFSL5 aregenerated based on the still images sampled when the camera movingdirection is the first camera moving direction DR1, and a positioningslit PFSL6 is generated based on the still image sampled when the cameramoving direction is the second camera moving direction DR2.

In this case, it is necessary to select the positioning slit used whensynthesizing the upper part (e.g., slit image SL5) and the lower part(e.g., slit image SL6) of the panoramic image. Since the third cameramoving direction DR3 is the downward direction in the exampleillustrated in FIG. 11B, the positioning slits PFSL4 and PFSL5 areassigned at the boundary between the upper half and the lower half ofthe panoramic image.

Therefore, the slit images are synthesized using the positioning slitsPFSL4 and PFSL5 generated based on the still images sampled when thecamera moving direction is the first camera moving direction DR1 in thesame manner as in the example illustrated in FIG. 11A.

FIG. 12A illustrates an example in which the positioning slit isgenerated from the lower part of the still image (i.e., a part of thestill image assigned in the direction opposite to the third cameramoving direction). In this case, positioning slits PFSL1 and PFSL2 aregenerated based on the still images sampled when the camera movingdirection is the first camera moving direction DR1, and a positioningslit PFSL3 is generated based on the still image sampled when the cameramoving direction is the second camera moving direction DR2.

In this case, it is necessary to select the positioning slit used whensynthesizing the upper part (e.g., slit image SL3) and the lower part(e.g., slit image SL2) of the panoramic image. Since the third cameramoving direction DR3 is the upward direction in the example illustratedin FIG. 12A, the positioning slit PFSL3 is assigned at the boundarybetween the upper half and the lower half of the panoramic image.

Therefore, in FIG. 12A, the slit images are synthesized using thepositioning slit PFSL3 generated based on the still image sampled whenthe camera moving direction is the second camera moving direction DR2.

FIG. 12B illustrates an example in which the positioning slit isgenerated from the upper part of the still image (i.e., a part of thestill image assigned in the direction opposite to the third cameramoving direction). In the example illustrated in FIG. 12B, positioningslits PFSL4 and PFSL5 are generated based on the still images sampledwhen the camera moving direction is the first camera moving directionDR1, and a positioning slit PFSL6 is generated based on the still imagesampled when the camera moving direction is the second camera movingdirection DR2.

In this case, it is necessary to select the positioning slit used whensynthesizing the upper part (e.g., slit image SL5) and the lower part(e.g., slit image SL6) of the panoramic image. Since the third cameramoving direction DR3 is the downward direction in the exampleillustrated in FIG. 12B, the positioning slit PFSL6 is assigned at theboundary between the upper half and the lower half of the panoramicimage.

Therefore, the slit images are synthesized using the positioning slitPFSL6 generated based on the still image sampled when the camera movingdirection is the second camera moving direction DR2.

As described above with reference to FIG. 8, it is necessary to performthe image comparison process while moving the slit image along thepositioning slit when specifying the overlapping area of the slit imagesusing the positioning slit.

A problem that may occur when using the above method is described belowwith reference to FIG. 13A. In FIG. 13A, a positioning slit PFSL1 and aslit image SL1 are generated from a single still image. A slit image SL2is generated from a different still image, and the overlapping area ofthe slit image SL2 and the positioning slit PFSL1 is specified. In thiscase, the slit image SL2 is assigned at the right end of the positioningslit, and the image comparison process is performed while moving theslit image SL2 in the leftward direction as described above withreference to FIG. 8, for example. When it has been determined that theimage included in the slit image SL2 coincides with the image includedin the positioning slit PFSL1, the coincident area is determined to bethe overlapping area of the slit image SL2 and the positioning slitPFSL1. When the image included in the slit image SL2 does not coincidewith the image included in the positioning slit PFSL1, the imagecomparison process is performed while further moving the slit image SL2in the leftward direction. When the slit image SL2 has reached the leftend of the positioning slit, the slit image SL2 is shifted in thedownward direction, and the image comparison process is performed whilemoving the slit image SL2 in the horizontal direction. According to theabove method, however, it is necessary to search the overlapping areaover the entire combinatorial area SA1 (shaded area) of the slit imageSL2 and the positioning slit PFSL1 (see FIG. 13A). Therefore, the loadof the process that specifies the overlapping area increases. Theprocess that specifies the overlapping area can be performed at highspeed if the overlapping area search range can be limited.

In order to deal with the above problem, when the imaging section ismoved as indicated by a motion vector DV1 (indicated by the motioninformation) (see FIG. 13B), the position of the overlapping area of theslit image SL4 is estimated based on the vertical component DVP1 and thehorizontal component DVH1 of the motion vector DV1, and the search rangeis limited to a search range SA2 (shaded area). In other words, theposition of the slit image SL4 after the imaging section has been movedis estimated based on the motion vector DV1 and the position of the slitimage SL3 before the imaging section is moved. Note that the searchrange SA2 may be enlarged by given pixels in the upward direction, thedownward direction, the rightward direction, and the leftward directiontaking account of a motion vector calculation error, for example.

However, estimation in the direction perpendicular to the camera movingdirection is not necessarily advantageous taking account of camerashake, the motion of the head, and the like. It may be impossible tospecify the overlapping area when camera shake or the like hastemporarily occurred to a large extent.

In order to deal with such a problem, the position of the slit image SL6(see FIG. 13C) after the imaging section has been moved may be estimatedusing only the horizontal component DVH2 of the motion vector DV2, andthe entire slit image may be set to be the search range (SA3) (shadedarea) in the vertical direction. In this case, the image comparisonprocess is performed on the slit image SL6 and the positioning slitPFSL3 while moving the slit image SL6 in the vertical direction withoutmoving the slit image SL6 in the horizontal direction. Note that thesearch range SA3 may be enlarged by given pixels in the upwarddirection, the downward direction, the rightward direction, and theleftward direction in the same manner as the search range SA2 takingaccount of a motion vector calculation error, for example.

The still image need not be sampled when it is determined that theoverlapping area cannot be specified from the motion information.

A method that controls the sampling process so that the overlapping areacan be specified is described below.

The overlapping area of the slit images becomes narrow if the imageacquisition rate (sampling rate) is constant even when the camera movingspeed (shooting speed) has increased. In such a case, the slit imagesmay not have an overlapping area. This makes it impossible to generatethe panoramic image. On the other hand, unnecessary frames are acquired,and the amount of image information unnecessarily increases if thesampling rate is constant even when the camera moving speed hasdecreased. In this case, it may be difficult to synthesize the slitimages. Moreover, since the number of slit images to be synthesizedincreases when unnecessary frames are acquired, the processing loadincreases.

In one embodiment of the invention, the sampling rate is increased whenthe camera moving speed has increased.

As illustrated in FIG. 14, when the width of the slit image is 70 px(pixels), and the minimum width of the overlapping area between theadjacent slits is 10 px, the center-to-center distance between theadjacent slit images must be 60 px or less.

For example, the imaging section captures an image at 30 fps (frames persecond), and the camera moving speed acquired from the motioninformation is 10 px/frame. Note that the unit “px/frame” refers to thenumber of pixels by which the pixels within an identical range of theobject move per frame. For example, “10 px/frame” means that the pixelswithin an identical range of the object move by 10 px per frame.

In this case, a minimum overlapping area can be obtained by setting thesampling rate to 5 fps (=30 fps×(10 px/60 px)) or more. When the cameramoving speed has increased to 15 px/frame, 30 fps×(15 px/60 px)=7.5 fps.Therefore, the slit images can be acquired while maintaining the widthof the overlapping area by setting the sampling rate to 8 fps or more.

This makes it possible to obtain the slit images having an overlappingarea necessary and sufficient for synthesizing the panoramic image. Whenthe shutter speed (frame rate) is increased and decreased correspondingto the camera moving speed, the object may be blurred when the framerate is insufficient. However, when the sampling rate is increased anddecreased while keeping the frame rate constant, the amount of data canbe reduced (i.e., unnecessary frames can be eliminated) while preventinga situation in which the object is blurred.

The width of the slit image may be increased in order to provide theoverlapping area when the camera moving speed has increased.

FIG. 15 illustrates an example in which the width of the slit image is70 px, the minimum width of the overlapping area of the adjacent slitsis 10 px, and the sampling rate can be increased only up to 5 fps (orthe maximum capacity of the imaging section is 5 fps). When the samplingrate is set to 5 fps, it is possible to deal with a camera moving speedof up to 60 px/frame since the maximum center-to-center distance is 60px (=70 px−10 px). However, when the camera moving speed has increasedto 70 px/frame, for example, it is impossible to provide the overlappingarea of the slits since the frame rate cannot be further increased. Inthis case, the width of the slit image may be set to 80 px (=70 px+10px) or more in order to secure 10 px or more on each end of the slitimage.

This makes it possible to provide an overlapping area even when thesampling rate cannot be increased due to limitations to the hardware orthe network.

4. Method

According to the above embodiments, the image processing system includesan image acquisition section 270 that consecutively acquires stillimages, a direction determination section 210 that determines a cameramoving direction, the camera moving direction being a moving directionof an imaging section 12 during capture, a slit image generation section250 that generates slit images based on the consecutively acquired stillimages; and a panoramic image generation section 220 that performs asynthesis process of the slit images to generate a panoramic image. Thedirection determination section 210 determines whether the camera movingdirection is a first camera moving direction or a second camera movingdirection that differs from the first camera moving direction when theimaging section 12 is moved during capture. The panoramic imagegeneration section 220 determines a synthesis position of a slit imageamong the slit images based on the camera moving direction when theconsecutively acquired still images were captured, and performs thesynthesis process of the slit image to generate the panoramic image.

According to the above configuration, when the imaging section is movedin a plurality of camera moving directions during capture, still imagesused to generate the slit images can be consecutively acquired, and thecurrent camera moving direction can be determined. The slit images canbe generated based on the consecutively acquired still images.

The expression “consecutively acquires still images” used herein whenthe imaging section captures a still image means acquiring still imagesconsecutively captured while consecutively moving the imaging section.The expression “consecutively acquires still images” used herein whenthe imaging section captures a movie means that the image acquisitionsection 270 samples and acquires still images from a movie capturedwhile consecutively moving the imaging section, or means acquiring stillimages sampled by another functional section (e.g., imaging section)from a movie captured while consecutively moving the imaging section.

The term “slit image” used herein refers to part or the entirety of astill image among the consecutively acquired still images that has agiven area. The term “slit image” used herein refers to an image used togenerate the panoramic image. For example, the slit image refers to thearea SL1 illustrated in FIG. 4A or the area SL2 illustrated in FIG. 4B.

The term “camera moving direction” used herein refers to the directionof a vector that connects two arbitrary different points along a paththat is drawn when moving the imaging section. For example, the cameramoving direction refers to the direction DR1, DR2, or DR3 illustrated inFIG. 5. The camera moving direction may be set in advance. When one ofthe horizontal component and the vertical component of the motion vectorindicated by the motion information about the imaging section acquiredby the motion information acquisition section (described later) iscontinuously larger than the other component for a given period, thedirection corresponding to the component that is larger than the othercomponent may be determined to be the camera moving direction.

An arbitrary camera moving direction is referred to as the first cameramoving direction, and an arbitrary camera moving direction that differsfrom the first camera moving direction is referred to as the secondcamera moving direction. Note that the camera moving direction is notlimited to the first camera moving direction and the second cameramoving direction. A camera moving direction other than the first cameramoving direction and the second camera moving direction may also beused.

According to the above configuration, the synthesis position of the slitimage can be determined based on the camera moving direction, and thesynthesis process can then be performed.

The term “synthesis process” used herein refers to a process thatgenerates a single image so that an image acquired as a result ofperforming the synthesis process overlaps a newly generated slit image.

The term “synthesis position” used herein refers to the position of theslit image relative to an image acquired as a result of performing thesynthesis process.

Therefore, it is possible to generate a panoramic image of which theangle of view is wider in a plurality of directions than that of apanoramic image generated by moving the imaging section in one cameramoving direction. It is also possible to generate a panoramic image thatcovers a wide range of the imaging area of the imaging section in thevertical direction and the horizontal direction while reducing adifference in perspective due to motion parallax by utilizing the slitimage as a material for generating the panoramic image.

The image processing system may further includes a positioning slitgeneration section 260 that generates a positioning slit based on astill image among the consecutively acquired still images. The panoramicimage generation section 220 may determine the synthesis position of theslit image based on the camera moving direction when the consecutivelyacquired still images were captured, and the positioning slit, and mayperform the synthesis process of the slit image to generate thepanoramic image.

The term “positioning slit” used herein refers to part or the entiretyof a still image among the consecutively acquired still images. The term“positioning slit” used herein refers to an image that partiallycoincides with the slit image, but does not perfectly coincide with theslit image. For example, the positioning slit refers to the area PFSL1illustrated in FIG. 7A, the area PFSL2 illustrated in FIG. 7B, or thearea PFSL3 illustrated in FIG. 7C.

According to the above configuration, since a problem does not occureven when one slit image has an overlapping area with a plurality ofslit images, the overlapping area can be relatively easily specified,for example. Moreover, since the slit images can be synthesized in thesecond camera moving direction even when the slit images have not beencompletely synthesized in the first camera moving direction, it ispossible to prevent a situation in which the process that synthesizesthe slit images in the first camera moving direction becomes abottleneck, and the panoramic image generation process can be performedat high speed, for example.

The positioning slit generation section 260 may generate the positioningslit so that a long side of the slit image is orthogonal to a long sideof the positioning slit.

The expression “a long side of the slit image is orthogonal to a longside of the positioning slit” refers to the relationship between theslit image SL1 (or SL2) and the positioning slit PFSL illustrated inFIG. 8, for example. Note that the long side of the slit image may beapproximately orthogonal to the long side of the positioning slit.

The positioning slit is generated by extracting (e.g., copying) a givenarea of the still image as the positioning slit. The positioning slitmay be generated by generating a new image of the positioning slit basedon the still image.

According to the above configuration, the overlapping area can bespecified so that the short side of the slit image coincides with thelong side of the positioning slit, for example. Therefore, since theprocess that specifies the overlapping area is less frequently performedat the boundary between the positioning slits, it is possible tosuppress a situation in which the boundary between the positioning slitsis discontinuous, and the overlapping area of the slit images cannot bespecified, for example.

The positioning slit generation section 260 may set a first area and asecond area based on a straight line that passes through a center of thestill image, may generate a first positioning slit from the first area,and may generate a second positioning slit from the second area.

The first area refers to one of the areas of the still image divided bythe straight line that passes through the center of the still image. Thesecond area refers to the other area of the still image. In FIG. 9A, thestill image SIM1 is divided by the straight line that passes through thecenter CP1 of the still image SIM1. The area SUB1 corresponds to thefirst area, and the area SUB2 corresponds to the second area. In FIG.9B, the area SUB3 corresponds to the first area, and the area SUB4corresponds to the second area.

The above configuration makes it possible to selectively use the firstpositioning slit or the second positioning slit when specifying theoverlapping area of the slit images, for example. Specifically, it ispossible to select the positioning slit corresponding to the cameramoving direction, and determine the overlapping area even when theimaging section is moved in an arbitrary direction, for example.

The positioning slit generation section 260 may change an extractionposition of the positioning slit within the still image based on motioninformation about the imaging section 12 acquired from a motioninformation acquisition section 16, and may generate the positioningslit.

The term “motion information” used herein refers to sensor informationor a motion vector acquired by the motion information acquisitionsection.

The above configuration makes it possible to extract the positioningslit from the same position as that of the previously used positioningslit when the position of the imaging section has changed to a largeextent (see FIG. 10), and efficiently perform the search process thatsearches the overlapping area of the synthesis target image and thepositioning slit, for example.

The panoramic image generation section 220 may shift the synthesisposition in a direction perpendicular to the first camera movingdirection when it has been determined that the camera moving directionis the second camera moving direction, and may perform the synthesisprocess so that part of the positioning slit overlaps part of the slitimage to generate the panoramic image.

Specifically, the slit image SL3 (see FIG. 11A) captured when theimaging section is moved in the second camera moving direction DR2 issynthesized at a position shifted in the direction DR3 perpendicular tothe first camera moving direction DR1 from the synthesis position of theslit image SL2 captured when the imaging section is moved in the firstcamera moving direction DR1. Note that the panoramic image generationsection 220 may shift the synthesis position of the slit image capturedwhen the imaging section is moved in the second camera moving directionin a direction approximately perpendicular to the first camera movingdirection.

The above configuration makes it possible to generate a panoramic imagethat covers a wide range of the imaging area in the vertical directionand the horizontal direction when the imaging section was moved along azigzag path, for example.

It is desirable that the overlapping area be an area of the slit imageassigned on the end of the slit image (e.g., an area equal to about 20%of the entire slit image). For example, when half or more of the slitimage is the overlapping area, the effects achieved by synthesizing theslit images are small since the angle of view of the synthesized imageis not sufficiently larger than the angle of view of the original slitimage.

Specifically, it is desirable to synthesize only an area of the slitimage assigned on the end of the slit image in order to improve theeffects achieved by synthesizing the slit images. Therefore, it isdesirable that the positioning slit used for the synthesis process ofthe slit image be an image corresponding to the boundary between theslit images.

Moreover, it may be difficult to specify the overlapping area of theslit images using the positioning slit depending on the positionalrelationship between the slit image and the positioning slit.

As described above, it is important to appropriately use the positioningslit in order to improve the effects achieved by synthesizing the slitimages.

The direction determination section 210 may determine the camera movingdirection when the imaging section 12 is moved in the first cameramoving direction, moved in a third camera moving direction that differsfrom the first camera moving direction and the second camera movingdirection, and then moved in the second camera moving direction duringcapture. The panoramic image generation section 220 may use thepositioning slit generated based on the still image acquired when thecamera moving direction is the first camera moving direction for thesynthesis process when the positioning slit was generated from an areaof the still image assigned in the third camera moving directionrelative to a center of the still image. The panoramic image generationsection 220 may also use the positioning slit generated based on thestill image acquired when the camera moving direction is the secondcamera moving direction for the synthesis process when the positioningslit was generated from an area of the still image assigned in adirection opposite to the third camera moving direction relative to thecenter of the still image.

The third camera moving direction refers to a direction that differsfrom the first camera moving direction and the second camera movingdirection. For example, the third camera moving direction is thedirection DR3 illustrated in FIG. 5.

According to the above method, it is possible to specify the positioningslit assigned at the boundary between the slit image captured when theimaging section is moved in the first camera moving direction and theslit image captured when the imaging section is moved in the secondcamera moving direction, for example.

This makes it possible to specify the positioning slit by which the slitimage can be synthesized so that the angle of view of the synthesizedimage is sufficiently larger than the angle of view of the original slitimage.

The panoramic image generation section 220 may limit a search range fora position of an overlapping area of the positioning slit and the slitimage based on motion information about the imaging section 12 acquiredfrom a motion information acquisition section 16, and may perform thesynthesis process.

The term “overlapping area” used herein refers to an area in which twoimages are made to overlap by the synthesis process. For example, theoverlapping area refers to the area REA illustrated in FIG. 6.

The above configuration makes it possible to limit the overlapping areasearch range to an area around the position estimated based the motioninformation after the imaging section has been moved, and perform theprocess that specifies the overlapping area at high speed (i.e., reducethe load of the process that specifies the overlapping area), forexample.

Note that estimation in the direction perpendicular to the camera movingdirection is not necessarily advantageous taking account of camerashake, the motion of the head, and the like. It may be impossible tospecify the overlapping area when camera shake or the like hastemporarily occurred to a large extent. Therefore, the panoramic imagegeneration section 220 may limit the search range for the position ofthe overlapping area of the positioning slit and the slit image based ona horizontal component or a vertical component of the motion informationabout the imaging section 12 acquired from the motion informationacquisition section 16, and may perform the synthesis process.

According to the above configuration, the search range can be limitedrelative to the positioning slit by utilizing the motion information inthe horizontal direction.

In general, a difference in perspective due to motion parallax increasesin an area of the still image in the camera moving direction and an areaof the still image in the direction opposite to the camera movingdirection (i.e., the right end and the left end of the still image, orthe upper end and the lower end of the still image). It is desirablethat the image used as a material for generating the panoramic image befree from a difference in perspective.

Therefore, the slit image generation section 250 may generate the slitimages that have a long side along a direction corresponding to ahorizontal component or a vertical component, whichever is smaller inabsolute value, of a vector that indicates the camera moving direction.

For example, when the camera moving direction is the direction DR1 inFIG. 4A, the slit image SL1 is generated that has the long side LL1along the direction DRP1 corresponding to the horizontal component orthe vertical component, whichever is smaller in absolute value, of thevector that indicates the direction DR1.

The above configuration makes it possible to generate a slit image thatdoes not include an area of the still image in the camera movingdirection and an area of the still image in the direction opposite tothe camera moving direction, and suppress the effects of a difference inperspective due to motion parallax, for example.

The slit image generation section 250 may generate the slit images thathave the long side along the direction corresponding to the horizontalcomponent or the vertical component, whichever is smaller in absolutevalue, of the vector that indicates the camera moving direction, andinclude a center of a still image among the consecutively acquired stillimages.

For example, when the camera moving direction is the direction DR2 inFIG. 4B, the slit image SL2 is generated that has the long side LL2along the direction DRP2 corresponding to the horizontal component orthe vertical component, whichever is smaller in absolute value, of thevector that indicates the direction DR2, and includes the center CP ofthe still image SIM2.

The above configuration makes it possible to generate a slit image thatdoes not include an area of the still image in the camera movingdirection and an area of the still image in the direction opposite tothe camera moving direction, and includes the center of the still image,and further suppress the effects of a difference in perspective due tomotion parallax, for example.

The panoramic image generation section 220 may perform the synthesisprocess of the slit image so that the slit image has an overlapping areawith the slit image that was used for a preceding synthesis process inthe camera moving direction when the slit image used for the precedingsynthesis process was captured to generate the panoramic image.

The above configuration makes it possible to perform the synthesisprocess while making the generated slit images overlap in the cameramotion direction, for example.

The direction determination section 210 may determine whether the cameramoving direction is the first camera moving direction, the second cameramoving direction, or a third camera moving direction that differs fromthe first camera moving direction and the second camera moving directionwhen the imaging section 12 is moved during capture.

The above configuration makes it possible to generate the panoramicimage when moving the imaging section along a zigzag path, for example.It is possible to generate a panoramic image over a range wider than theangle of view of the imaging section in a plurality of directions bymoving the imaging section along a zigzag path during capture, forexample. Moreover, since it is unnecessary to move the imaging sectionfor positioning that adjusts the capture start position in the firstcamera moving direction and the capture start position in the secondcamera moving direction, it is possible to reduce the burden imposed onthe user, for example. Since the materials for generating the panoramicimage can be provided by capturing a movie once, it is possible toreduce the burden imposed on the user, for example.

The panoramic image generation section 220 may perform a process thatsynthesizes the slit images in the first camera moving direction and aprocess that synthesizes the slit images in the second camera movingdirection in parallel.

According to the above configuration, since the slit images can besynthesized in the second camera moving direction even when the slitimages have not been completely synthesized in the first camera movingdirection, it is possible to prevent a situation in which the processthat synthesizes the slit images in the first camera moving directionbecomes a bottleneck, and the panoramic image generation process can beperformed at high speed, for example.

The panoramic image generation section 220 may perform the synthesisprocess of the slit image based on a first error tolerance, and then mayperform the synthesis process of the slit image based on a second errortolerance that is lower than the first error tolerance.

The term “error tolerance” used herein refers to a degree by which ashift from the target position of the imaging section is allowed. Theterm “target position” used herein refers to the position of the imagingsection when the imaging section has been moved from the currentposition in the direction opposite to the direction of the vectorindicated by the camera motion direction information (acquired when theimaging section was moved in the first camera moving direction) by anidentical distance.

The term “error tolerance” used herein may refer to the tolerance forthe ratio of inconsistency of the overlapping area of two slit images.Specific examples of the error tolerance include the tolerance for thedifference in position of the pixels within an identical range of theobject, and the like.

The first error tolerance refers to a given error tolerance that ishigher than the second error tolerance. The second error tolerancerefers to a given error tolerance that is lower than the first errortolerance.

According to the above configuration, the overlapping area of the slitimage can be specified based on the first error tolerance, and theaccuracy of the synthesis position of the slit image can be improvedbased on the second error tolerance, for example. This makes it possibleto generate a panoramic image in which two slit images are made tooverlap so that the difference in position of the pixels within anidentical range of the object decreases, for example.

The image acquisition section 270 may stop acquisition of the stillimages when motion information about the imaging section 12 has beenacquired from motion information acquisition section 16, and it has beendetermined that an overlapping area of consecutive slit images among theslit images has not occurred in the synthesis process of the slit imagebased on the motion information.

The above configuration makes it possible to prevent a situation inwhich the still images are consecutively acquired although consecutiveslit images do not have an overlapping area, and a panoramic imagecannot be generated, for example.

The image acquisition section 270 may increase an image acquisition rateso that an overlapping area of consecutive slit images among the slitimages is larger than a given threshold value during the synthesisprocess of the slit image when it has been determined that a cameramoving speed has increased based on motion information about the imagingsection 12 acquired from a motion information acquisition section 16,the camera moving speed being a moving speed of the imaging section 12.

The term “camera moving speed” used herein refers to the moving speed ofthe imaging section that is moved by the user.

The term “image acquisition rate” used herein refers to the number ofstill images acquired per unit time. Specifically, the image acquisitionrate refers to the same rate as the shutter speed of the imagingsection, a given rate lower than the shutter speed of the imagingsection, or the like when the I/F section acquires information includingconsecutive still images, and refers to the sampling rate of stillimages from a movie or the like when the I/F section acquiresinformation including a movie.

The above configuration makes it possible to obtain the slit imageshaving an overlapping area necessary for synthesizing the panoramicimage.

The image acquisition section 270 may increase the image acquisitionrate during the synthesis process performed on the slit image when ithas been determined that the camera moving speed (i.e., the moving speedof the imaging section 12) has increased based on the motion informationabout the imaging section 12 acquired from the motion informationacquisition section 16.

For example, when it has been determined that the camera moving speedhas increased, the image acquisition rate may be changed to a presetimage acquisition rate (sampling rate) that is higher than a normalimage acquisition rate (sampling rate).

The slit image generation section 250 may increase a width of the slitimage so that an overlapping area of consecutive slit images among theslit images is larger than a given threshold value during the synthesisprocess of the slit image when it has been determined that a cameramoving speed has increased based on motion information about the imagingsection 12 acquired from a motion information acquisition section 16,the camera moving speed being a moving speed of the imaging section 12.

The above configuration makes it possible to obtain the slit imageshaving an overlapping area necessary for synthesizing the panoramicimage even when the image acquisition rate (sampling rate) cannot beincreased due to limitations to the hardware or the network, forexample.

The slit image generation section 250 may increase the width of the slitimage when it has been determined that the camera moving speed (i.e.,the moving speed of the imaging section 12) has increased based on themotion information about the imaging section 12 acquired from the motioninformation acquisition section 16.

For example, when it has been determined that the camera moving speedhas increased, the width of the slit image may be changed to a presetslit width that is larger than a normal slit width.

When the still image capture rate achieved by the imaging device 100 (orthe sampling rate of still images from a movie achieved by the imagingdevice 100 (hereinafter the same)) is higher than the still imageacquisition rate achieved by the image acquisition section 270, some ofthe still images acquired by the imaging device 100 are wasted. When thestill image capture rate achieved by the imaging device 100 is lowerthan the still image acquisition rate achieved by the image acquisitionsection 270, the image acquisition section 270 cannot acquire asufficient number of images necessary for generating the panoramicimage.

The image acquisition section 270 may control a shutter release timingof the imaging section to consecutively acquire the still images.

The shutter release timing is controlled by outputting information aboutthe sampling rate set by the sampling control section 272 or the like tothe imaging device 100. Information about the shutter speed, theshooting (capture) interval, or the like calculated based on thesampling rate may also be output to the imaging device 100.

The above configuration makes it possible to cause the number of stillimages captured by the imaging section to coincide with the number ofstill images acquired by the image acquisition section, and cause theimage acquisition section to acquire the desired number of still imageswhile preventing a situation in which the imaging section capturesunnecessary still images, for example.

The image processing system, the information processing device, and thelike according to the embodiments of the invention may be implemented bya program or a computer-readable information storage device that storesthe program. In this case, the image processing system, the informationprocessing device, and the like according to the embodiments of theinvention are implemented by causing a processor (e.g., CPU) to executethe program. Specifically, the program stored in the information storagedevice is read from the information storage device, and the processor(e.g., CPU) executes the program read from the information storagedevice. The information storage device (computer-readable device) storesa program, data, and the like. The function of the information storagedevice may be implemented by an optical disk (e.g., DVD or CD), a harddisk drive (HDD), a memory (e.g., memory card or ROM), or the like. Theprocessor (e.g., CPU) performs various processes according to theembodiments of the invention based on the program (data) stored in theinformation storage device. Specifically, a program that causes acomputer (i.e., a device including an operation section, a processingsection, a storage section, and an output section) to function as eachsection according to the embodiments of the invention (i.e., a programthat causes a computer to execute the process implemented by eachsection) is stored in the information storage device.

The image processing system, the information processing device, and thelike according to the embodiments of the invention may include aprocessor and a memory. The processor may be a central processing unit(CPU), for example. Note that the processor is not limited to a CPU.Various types of processors such as a graphics processing unit (GPU) anda digital signal processor (DSP) may also be used. The processor may bea hardware circuit that includes an application specific integratedcircuit (ASIC). The memory stores a computer-readable instruction. Eachsection of the image processing system, the information processingdevice, and the like according to the embodiments of the invention isimplemented by causing the processor to execute an instruction. Thememory may be a semiconductor memory (e.g., static random access memory(SRAM) or dynamic random access memory (DRAM)), a register, a hard disk,or the like. The instruction may be an instruction included in aninstruction set of a program, or may be an instruction that causes ahardware circuit of the processor to operate.

5. Flow of Process

The flow of the process according to the embodiments of the invention isdescribed below using the flowcharts illustrated in FIGS. 16 and 17.Note that the flow of the process is described below taking an examplein which the panoramic image is generated from a movie. However, thepanoramic image need not necessarily be generated from a movie. Thepanoramic image can also be generated in the same manner as describedbelow when generating the panoramic image from consecutively capturedstill images.

5.1 Flow of Panoramic Image Generation Process

In a step S1, the user starts capturing a movie. The motion informationacquisition section then acquires the motion information (S2). Whetheror not the imaging section is being moved in a specific direction isdetermined based on the acquired motion information (S3). When it hasbeen determined that the imaging section is not being moved in aspecific direction, the step S2 is performed again. When it has beendetermined that the imaging section is being moved in a specificdirection, the camera moving speed is calculated based on the motioninformation, and the sampling rate is set based on the camera movingspeed (S4). Still images are sampled from the captured movie based onthe sampling rate set in the step S4 (S5).

An optimum slit image width is set based on the motion information sothat an overlapping area occurred (S6). A slit image having the slitimage width set in the step S6 is extracted from the sampled still image(S7). A positioning slit is generated from the still image (S8).

Whether or not the imaging section is being moved is determined based onthe motion information (S9). When it has been determined that theimaging section is being moved, the step S7 is performed again. When ithas been determined that the imaging section is not being moved, thestill image sampling process is terminated (S10). The panoramicsynthesis process is then performed to generate a panoramic image (S11).The details of the panoramic synthesis process are described below.

5.1.1 Flow of Panoramic Synthesis Process

The first error tolerance range is set in a step S20. The overlappingarea search range is limited using the motion information (S21).

The overlapping area of the positioning slit and the slit image capturedwhen the imaging section is moved in the first camera moving directionis calculated (S22). The overlapping area of the positioning slit andthe slit image captured when the imaging section is moved in the secondcamera moving direction is calculated in parallel with the step S22(S22). The slit image is synthesized based on the first error tolerancerange (S24).

Whether or not all of the slit images have been synthesized isdetermined (S25). When it has been determined that all of the slitimages have not been synthesized, the step S21 is performed again. Whenit has been determined that all of the slit images have beensynthesized, the second error tolerance range is set (S26). The accuracyof the synthesis position of the slit image is improved based on thesecond error tolerance range set in the step S26 (S27).

Although only some embodiments of the invention have been described indetail above, those skilled in the art will readily appreciate that manymodifications are possible in the embodiments without materiallydeparting from the novel teachings and advantages of the invention.Accordingly, all such modifications are intended to be included withinscope of the invention. Any term cited with a different term having abroader meaning or the same meaning at least once in the specificationand the drawings can be replaced by the different term in any place inthe specification and the drawings. The configuration and the operationof the image processing system, the information processing device, andthe information storage device are not limited to those described inconnection with the above embodiments. Various modifications andvariations may be made of the above embodiments.

What is claimed is:
 1. An image processing system comprising: aprocessor comprising hardware, wherein the processor is configured to:acquire a plurality of still images captured by an image sensor, whereina first still image of the plurality of still images is captured as theimage sensor is moved in a first moving direction, and a second stillimage of the plurality of still images is captured as the image sensoris moved in a second moving direction different from the first movingdirection; extract a first silt image from the first still image,wherein the first slit image has a long side along a first slitdirection; extract a second slit image from the second still image,wherein the second slit image has a long side along a second slitdirection; extract a positioning slit from one of the plurality of stillimages, wherein the positioning slit has a long side along a third slitdirection different from the first slit direction and the second slitdirection; determine a first overlapping area between the first slitimage and the positioning slit; determine a second overlapping areabetween the second slit image and the positioning slit; determine athird overlapping area between the first overlapping area and the secondoverlapping area; and generate a panoramic image based on the first slitimage and the second slit image for which the third overlapping area hasbeen determined.
 2. A computer-readable storage device with anexecutable program stored thereon, wherein the program instructs acomputer to perform steps of: acquiring a plurality of still imagescaptured by an image sensor, wherein a first still image of theplurality of still images is captured as the image sensor is moved in afirst moving direction, and a second still image of the plurality ofstill images is captured as the image sensor is moved in a second movingdirection different from the first moving direction; extracting a firstsilt image from the first still image, wherein the first slit image hasa long side along a first slit direction; extracting a second slit imagefrom the second still image, wherein the second slit image has a longside along a second slit direction; extracting a positioning slit fromone of the plurality of still images, wherein the positioning slit has along side along a third slit direction different from the first slitdirection and the second slit direction; determining a first overlappingarea between the first slit image and the positioning slit; determininga second overlapping area between the second slit image and thepositioning slit; determining a third overlapping area between the firstoverlapping area and the second overlapping area; and generating apanoramic image based on the first slit image and the second slit imagefor which a third overlapping area has been determined.
 3. An imageprocessing method comprising: acquiring a plurality of still imagescaptured by an image sensor, wherein a first still image of theplurality of still images is captured as the image sensor is moved in afirst moving direction, and a second still image of the plurality ofstill images is captured as the image sensor is moved in a second movingdirection different from the first moving direction; extracting a firstsilt image from the first still image, wherein the first slit image hasa long side along a first slit direction; extracting a second slit imagefrom the second still image, wherein the second slit image has a longside along a second slit direction; extracting a positioning slit fromone of the plurality of still images, wherein the positioning slit has along side along a third slit direction different from the first slitdirection and the second slit direction; determining a first overlappingarea between the first slit image and the positioning slit; determininga second overlapping area between the second slit image and thepositioning slit; determining a third overlapping area between the firstoverlapping area and the second overlapping area; and generating apanoramic image based on the first slit image and the second slit imagefor which a third overlapping area has been determined.